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The findings from this review do not reflect the full body of research evidence 
on Problem Based Economics Instruction. 


What is this study about? 

The study included 128 high school economics teach- 
ers from 106 schools in Arizona and California, half of 
whom were randomly assigned to the Problem Based 
Economics Instruction condition and half of whom were 
randomly assigned to the comparison condition. High 
levels of teacher attrition occurred after randomiza- 
tion and before implementation. The analysis sample 
included 64 teachers, with 35 in the treatment condition 
and 29 in the comparison condition. 3 Student attrition 
was low, and the student analytic sample was shown 
to be equivalent in economic literacy at baseline. 4 

Intervention teachers used Problem Based Economics 
Instruction materials as a major portion of their cur- 
riculum content and instructional program during the 
2007-08 academic year, whereas comparison teachers 
used their schools’ standard instructional materials. 

The study assessed the effectiveness of Problem Based 
Economics Instruction by comparing the economics 
knowledge of students in the treatment and comparison 
groups at the end of their implementation semester. 


WWC Rating 


The research described in this 
report meets WWC evidence 
standards with reservations 

Strengths: This study is a well-implemented 
randomized controlled trial. 

Cautions: This study experienced high levels of 
teacher attrition. 


Features of Problem Based Economics Instruction 


Building on the more general techniques of problem- 
based learning, the Problem Based Economics 
Instruction curriculum was designed by the Buck 
Institute for Education, with input from university 
economists and expert teachers. The intervention is 
intended to help students actively learn critical thinking 
and problem-solving skills using real-world examples. 

Each economics unit took place over four to 15 
instructional days. Teachers were asked to provide 
core course content and use a set of strategies 
to help students contextualize, comprehend, and 
solve real economic problems; work in a group; 
communicate effectively using multiple methods and 
technologies; gather information and analyze data; 
understand interrelationships across economics 
systems; and make inferences. 

The curriculum was designed to include nine modules. 
Five of the nine available modules were selected for 
use in this study and were provided to the intervention 
group teachers; these modules were chosen because 
they included fundamental components of the 
curriculum standards in Arizona and California. 

What did the study find? 5 

The study found no statistically significant difference 
on the economics knowledge of students in grades 
1 1 and 12 in the classes that used Problem Based 
Economics Instruction, relative to students in the 
comparison classes. However, the effect size of 0.29 
for the economics knowledge domain was positive 
and large enough to be considered substantively 
important according to WWC criteria (that is, at least 
0.25 standard deviations). 
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Appendix A: Study details 

Finkelstein, N., Hanson, T., Huang, C.-W., Hirschman, B., & Huang, M. (2011). Effects of Problem Based 
Economics on high school economics instruction (NCEE 2010-4002rev). Washington, DC: National 
Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. 
Department of Education. 


Setting 

The study was conducted in 106 high schools in Arizona and California. 

Study sample 

The original study sample included 128 teachers from 106 schools in Arizona and California 
who were expected to teach high school economics classes for two consecutive semesters. 
The teachers were randomly assigned to either the Problem Based Economics Instruction 
condition or the comparison condition, resulting in 64 teachers in each group. For the 16 
schools with two or more teachers in the study sample, randomization was conducted within 
the school. For other schools, randomization was conducted within strata of schools defined 
by their 2006 average student achievement test results. Students in grades 1 1 and 12 selected 
their economics classes around the same time the teachers received their assignments. There 
was no indication that students’ course selection was related to teacher assignment. 

Half of the teachers left or were dropped from the study sample after random assignment, 
resulting in an analysis sample of 64 teachers (35 in the Problem Based Economics Instruction 
group and 29 in the comparison group). 6 Attrition among students of teachers in the analysis 
sample was very low. The analytic sample (defined as those students in the analysis sample 
with no missing data) included 1,728 students in 35 treatment classrooms and 1,150 students 
in 29 comparison classrooms in the spring 2008 semester. 7 Across all 106 schools in the analy- 
sis sample, 60% of the students were from minority race/ethnic groups, and 39% received free 
or reduced-price lunches. Eighty-eight percent of students with valid posttest measures were 
enrolled in grade 1 2; the remaining 1 2% of students in the analysis sample were in grade 1 1 . 

Intervention 

group 

Teachers in intervention classrooms used Problem Based Economics Instruction materials as 
a major portion of their curriculum content and instructional program. Teachers implemented 
the one-semester curriculum to one set of students in the fall of 2007 and to a second set of 
students in the spring of 2008. The analytic sample only included students from the spring 
semester. Five of the nine available modules were selected for use in this study, as these mod- 
ules address the fundamental concepts required for the economics curriculum in Arizona and 
California. Use of Problem Based Economics Instruction represented approximately 50%-70% 
of the instruction received by students in the intervention condition. 

Comparison 

group 

Teachers in the comparison group used their standard instructional practices during both 
semesters. They received regular professional development in the 2007-08 academic year. 

Outcomes and 
measurement 

After completing one semester of high school economics (in spring 2008), students completed 
the Test of Economic Literacy and a performance task assessment that measured their conceptual 
knowledge and economic problem-solving skills at the end of the spring 2008 semester. For a 
more detailed description of these outcome measures, see Appendix B. 
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Support for 
implementation 


Reason for 
review 


In the summer prior to implementation, intervention teachers received 40 hours of professional 
development in Problem Based Economics Instruction over the course of five days. The trainers 
reviewed one curriculum module each day. Teachers also received five additional support ses- 
sions through group phone conferences. During these phone conferences, teachers discussed 
the curriculum and came up with solutions to challenging questions they encountered in class. 
Teachers also had access to master teachers and developer staff (from Buck Institute for Edu- 
cation) if they had questions about program implementation. 

This study was identified for review by the WWC because it is an Institute of Education Sciences 
(lES)-funded study conducted by 2006-1 1 Regional Education Laboratory West administered 
by WestEd. 
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Appendix B: Outcome measures for the economics knowledge domain 


Economics 


Economics 


Test of Economic Literacy 

The Test ot Economic Literacy, a nationally normed standardized achievement test, was developed by the 
National Council on Economic Education to assess basic economic concepts taught in high school economics 
courses in grades 11 and 12. This test has two forms, each including 40 multiple-choice items. Eleven items 
overlap across the two forms. Students are required to finish the test in 30 to 40 minutes. This assessment 
was used as a pretest and posttest measure in the current study, with form A serving as the pretest and form B 
serving as the posttest. 

Student Performance Task Assessment 

This test, developed by the National Center for Research on Education, Standards, and Student Testing at the 
University of California, Los Angeles (UCLA CRESST), was used as a posttest assessment of student conceptual 
knowledge and economic problem-solving skills. The test consisted of five assessment tasks that focused on 
federal funds policy, employment policy, fiscal policy, consumer demand, and opportunity costs. These tasks 
were chosen because of their focus on fundamental economics concepts and alignment with state standards 
for high school economics courses. Students in the current study were randomly assigned to complete two of 
the five assessment tasks at posttest. Responses were evaluated by Educational Data Systems, Inc., on overall 
content understanding, prior knowledge, number of principles or concepts used, use of text for elaboration, 
quality of the argument, and misconceptions in interpretation of the text. 
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Appendix C: Study findings for the economics knowledge domain 





Mean 

(standard deviation) 

WWC calculations 


Domain and outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Economics knowledge 

Test of Economic Literacy 

Grades 
11 and 12 

64 teachers/ 
2,878 students 

23.29 

(7.88) 

20.55 

(8.15) 

2.74 

0.34 

+13 

0.03 

Student Performance Task 
Assessment 

Grades 
11 and 12 

62 teachers/ 
2,657 students 

6.78 

(2.14) 

6.29 

(2.01) 

0.49 

0.24 

+9 

0.06 

Domain average for economics knowledge 





0.29 

+11 

Not 

statistically 

significant 


Table Notes: Positive results for mean difference, effect size, and improvement index favor the intervention group; negative results favor the comparison group. The effect size is 
a standardized measure of the effect of an intervention on student outcomes, representing the change (measured in standard deviations) in an average student's outcome that can 
be expected if the student is given the intervention. The improvement index is an alternate presentation of the effect size, reflecting the change in an average student's percen- 
tile rank that can be expected if the student is given the intervention. The WWC-computed average effect size is a simple average rounded to two decimal places; the average 
improvement index is calculated from the average effect size. The statistical significance of the study's domain average was determined by the WWC; a study is characterized as 
not statistically significant when univariate statistical tests are reported for each outcome measure and no effects within the domain are statistically significant. 

Study Notes: The adjusted posttest means presented here were reported in an appendix of the original study and come from multilevel regression models that accounted for dif- 
ferences in study design characteristics, teacher characteristics, student demographic characteristics, and student pretest data from the Test of Economic Literacy. The analytical 
model represented here included only students with nonmissing posttest and covariate data. The p-values presented here were reported in the original study. A correction for 
multiple comparisons was needed and resulted in significance levels that differ from those in the original study. Due to the multiple comparisons adjustment, the p-value of 0.03 
for the Test of Economic Literacy was higher than the critical p-value for statistical significance; therefore, the WWC does not find the result to be statistically significant. 
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Endnotes 

1 Single study reviews examine evidence published in a study (supplemented, if necessary, by information obtained directly from the 
authors]) to assess whether its design meets WWC evidence standards. The review reports the WWC’s assessment of whether the 
study meets WWC evidence standards and summarizes the study findings following WWC conventions for reporting evidence on 
effectiveness. The WWC rating applies only to the summarized results, and not necessarily to all results presented in the study. This 
study was reviewed using the Single Study review protocol. 

2 Absence of conflict of interest: The Regional Educational Labs were provided technical assistance by Mathematica Policy Research, 
which also operates the WWC. For this reason, this study was reviewed by staff from subcontractor organizations. 

3 These numbers reflect the analysis sample for the Test of Economic Literacy. The Student Performance Task Assessment analysis 
included 62 teachers (33 in the Problem Based Economics Instruction group and 29 in the comparison group). 

4 The authors provided the WWC with sample sizes, means, and standard deviations for the Test of Economic Literacy at baseline for 
the sample of students without imputed data. 

5 Finkelstein et al. (2011) found a statistically significant difference between the treatment and comparison groups on the Test of Eco- 
nomic Literacy measure and on the Student Performance Task Assessment. These effects were not statistically significant according 
to the WWC criteria, however, when using non-imputed data and correcting for multiple comparisons. 

6 Teachers left or were dropped from the study for various reasons, such as class schedule and position changes, personal issues, 
unavailability for training, and unresponsiveness to study contact attempts. 

7 These numbers reflect the analysis sample (defined as those students in the analysis sample with no missing data) for the Test of 
Economic Literacy. The Student Performance Task Assessment included 2,657 students (1 ,551 in the Problem Based Economics 
Instruction group and 1,106 in the comparison group). 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2012, June). WWC 
review of the report: Effects of Problem Based Economics on high school economics instruction. Retrieved 
from http://whatworks.ed.gov. 
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Glossary of Terms 

Attrition 


Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Improvement index 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Single-case design 
(SCD) 

Standard deviation 


Statistical significance 
Substantively important 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review and inclusion in this report if it falls within the scope of the 
review protocol and uses either an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

Along a percentile distribution of students, the improvement index represents the gain 
or loss of the average student due to the intervention. As the average student starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which subjects are assigned 
to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which investigators randomly assign 
eligible participants into intervention and comparison groups. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample tend to be spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% Ip < 0.05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 2.1) for additional details. 
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